Many businesses and organizations use complex, network distributed applications. These applications may be employed as part of the internal operation of the business, as a product or service offered to customers, or as part of a larger system involving other businesses and organizations. Distributed applications often have complex topologies that include remote data-centers, third party services, cloud-based components, mobile and Web-based interfaces, and globalized infrastructure. These applications often have multiple distinct functional tiers, implement a range of different protocols, and use hardware and software from a variety of different vendors.
In order to manage and maintain complex distributed applications, different strategies have been employed. According to one strategy, network communications between nodes are monitored. Exchanges of data and/or messages characterize the individual nodes and their relationship to each other.
Depending on the type of application, a particular exchange of data may correspond to the execution of a particular task. Such an exchange of data may be identified as a type of transaction. Monitoring and analyzing the transactions between nodes provides critical information about the operation and performance of the individual nodes and of the overall distributed application.
One method of monitoring inter-node communications involves sending a copy of all related network traffic to an analysis system. For example, network traffic may be monitored by connecting an analysis system to a “mirror port” of a network switch. A mirror port is a special interface to which the switch sends a copy of every packet passing through the other interfaces of the switch. An analysis system may be connected to the mirror ports on multiple switches in order to acquire copies of all network packets that are transmitted in the operation of the monitored application.
The analysis system may operate off-line on a recorded trace of network packets or it may process the packets in real-time as they arrive. The analysis system processes network packets to extract the application messages that were sent between nodes. The application messages are further analyzed to determine the type and nature of transactions that were executed between individual nodes. Transactions between nodes may be further analyzed to associate them with transactions between other nodes. In a further analysis, transactions may be associated with actions of an end-user of the application and with the business processes that they are intended to support.
For example, a basic transaction between two individual nodes may correspond to the update of data within a database. A set of similar transactions may correspond to an end-user making changes to personal data maintained by an application. Several sets of transactions may correspond to the end-user managing a bank account including updating personal data, transferring money, and paying bills. The analysis system decodes network data, extracts the messages, constructs individual transactions, and successively groups them together and characterizes them with respect to application design and established business processes. The results of these analyses are presented to a user of the analysis system so that they may be viewed and measured. From this information, a user such as an application manager or customer support technician may determine the overall behaviour of the application with respect to a business process in order to confirm correct operation, diagnose problems, or optimize performance. As well, individual transactions corresponding to specific incidents or customer actions may be isolated and inspected.
Networks and applications are implemented using various well-known protocols, conventions, and programming patterns. However, the inter-play between the application components, the underlying network, and the software and hardware environments leads to complexities in application behaviour. These behaviours are often not fully anticipated by the mechanics of the implemented protocols or by the application designers. One shortcoming of existing analysis systems is that their simple rule-based analysis of network traffic is often not sufficient to accurately and consistently identify and correlate transactions in modern distributed applications. Rather, semantic analysis, using domain knowledge, is required to effectively monitor business transactions. Efficient means are required to implement the necessary models to analyze network traffic using semantic analysis. Such means are often lacking in existing systems.
For a transaction monitoring system to operate in real-time on network traffic from a large distributed application, very high volumes of network data must be processed. Real-time analysis must efficiently operate on the network data without losing information, generating incorrect or incomplete results, exceeding the capacity of the resources being used for monitoring, or impacting the application that is being monitored. Efficient mechanisms are required for data monitoring, recording, analysis, storage and retrieval. Again, such mechanisms are often lacking in existing systems.
In general, real-time analysis systems are designed to minimize bottlenecks and optimize the use of resources to satisfy specific temporal constraints. In the case of transaction monitoring, network traffic must be successfully assembled into transactions at a rate higher than new network data arrives in order to avoid losing yet-unprocessed data or exceeding data buffer limitations. Existing database storage mechanisms are not sufficiently time-efficient to support high volumes of data for processing by a real-time system. Conversely, centralized data processing that requires data to be moved from storage to a processing engine for analysis creates a bottleneck. Further, analyzing all data without pre-processing, partitioning and filtering is highly inefficient. As such, existing methods and systems for real-time transaction monitoring based upon network traffic are not efficient enough to meet modern needs.
A need therefore exists for an improved method and system for generating transaction data from network traffic data for an application system which is distributed across a plurality of network connected nodes. Accordingly, a solution that addresses, at least in part, the above and other shortcomings is desired.